Skip to content

Implement way to run pipeline commands locally in docker, starting with fetch#5

Closed
Staberinde wants to merge 5 commits intomainfrom
run-pipeline-commands-locally
Closed

Implement way to run pipeline commands locally in docker, starting with fetch#5
Staberinde wants to merge 5 commits intomainfrom
run-pipeline-commands-locally

Conversation

@Staberinde
Copy link
Copy Markdown
Contributor

@Staberinde Staberinde commented Jan 7, 2022 •
edited
Loading

TLDR
when you run this make command from a collection repo like so:

 make ENDPOINT_URL='<url goes here>' dockerised-fetch

It creates a local_collection folder within your collection repo and fetches the resource as the pipeline would, and puts the output in local_collection in the same format as the pipeline would.
Output:

mkdir -p local_collection

2022-01-07 17:47:55,997 INFO get https://maps.buckscc.gov.uk/arcgis/rest/services/PLANNING/RIPA_BOPS/MapServer/2/query?where=1%3D1&text=&objectIds=&time=&geometry=&geometryType=esriGeometryPolygon&inSR=&spatialRel=esriSpatialRelIntersects&relationParam=&outFields=*&returnGeometry=true&returnTrueCurves=false&maxAllowableOffset=&geometryPrecision=&outSR=&returnIdsOnly=false&returnCountOnly=false&orderByFields=&groupByFieldsForStatistics=&outStatistics=&returnZ=false&returnM=false&gdbVersion=&returnDistinctValues=false&resultOffset=&resultRecordCount=&f=geojson
2022-01-07 17:47:58,082 INFO collection/resource/e27ba5fb720eea44ad4d7f34a720d54303affc80faaa9e2874fe77dacfaf4b02
2022-01-07 17:47:58,085 INFO collection/log/2022-01-07/8ce86423c7426509901eb2dd87a0207f6b442c074a4336fe41b818dc02115242.json

Result:

> tree local_collection/
local_collection/
└── collection
    ├── log
    │   └── 2022-01-07
    │       └── 8ce86423c7426509901eb2dd87a0207f6b442c074a4336fe41b818dc02115242.json
    └── resource
        └── e27ba5fb720eea44ad4d7f34a720d54303affc80faaa9e2874fe77dacfaf4b02

Everything runs in docker so all the dependencies are taken care of (it will even pull the image once I set the ECR up)
This should help us debug issues with the pipeline in a much easier way.

TODO

  • Check weird issue if fixed where collection/pipeline.mk is deleted when running against collection repo (specifically listed-building-collection)

@Staberinde Staberinde changed the title Run pipeline commands locally Implement way to run pipeline commands locally in docker, starting with fetch Jan 7, 2022
@Staberinde Staberinde force-pushed the run-pipeline-commands-locally branch from d55395e to 8256996 Compare January 10, 2022 12:07
@Staberinde Staberinde mentioned this pull request Jan 10, 2022
Closed
1 task
@Staberinde
Copy link
Copy Markdown
Contributor Author

Staberinde commented Aug 17, 2022

Closing, as deprecated by #7

@Staberinde Staberinde closed this Aug 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Staberinde